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PREFACE 


This  work  was  performed  for  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness,  Readiness  and  Training  Directorate,  under  a  task  entitled  “Development  and 
Assessment  of  ADL  Prototypes.” 

Despite  the  many  common  goals  between  the  development  of  Advanced  Distributed 
Learning  (ADL)  sharable  instructional  objects  and  intelligent  tutoring  systems  (ITSs),  this 
development  is  being  pursued  in  parallel  but  independently.  This  document  is  intended  to 
promote  more  joint  effort  between  these  two  development  communities  and  show  why  such 
joint  development  will  benefit  the  Department  of  Defense  (DoD)  by  reducing  training  and 
development  costs  and  increasing  the  effectiveness  of  its  training,  education,  and  decision- 
aiding  activities. 

An  earlier  version  of  this  document  was  published  in  the  International  Journal  of 
Cognitive  Ergonomics  and  was  reviewed  by  its  editors  and  two  anonymous  reviewers. 
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EXECUTIVE  SUMMARY 


The  techniques  of  object-oriented  programming  are  influencing  many  areas  of  soft¬ 
ware  development,  including  the  development  of  tools,  materials,  and  applications  for 
technology-based  education  and  training.  Reviews  of  technology-based  instruction  across 
many  subject  matters  and  settings  have  shown  that  it  reduces  costs  by  about  a  third  and  also 
either  reduces  the  time  to  reach  given  instructional  objectives  by  about  a  third  or  increases 
the  achievement  of  students  by  about  a  third.  An  important  benefit  that  technology-based 
instruction  provides  for  mihtary  and  industrial  training  is  accessibility.  It  can  be  used 
effectively  anytime  and  anywhere. 

These  capabilities  have  motivated  the  Department  of  Defense  (DoD),  in  cooperation 
with  many  other  Federal  agencies,  to  undertake  an  initiative  in  Advanced  Distributed 
Learning  (ADL).  ADL  combines  the  economic  efficiencies  of  object-oriented  instruction 
(which  provides  reusable,  sharable  instructional  objects)  and  the  learning  efficiencies  and 
accessibility  of  technology-based  instruction.  ADL  is  also  targeting  a  third  set  of  capabili¬ 
ties,  namely  those  of  intelligent  tutoring  systems  (ITSs). 

ADL’s  intention  is  to  allow  the  real-time  assembly,  or  generation,  of  instmctional 
objects  into  instructional  or  decision-aiding  presentations  on  demand  (as  needed  by  indi¬ 
vidual  learners  and  users).  It  expects  to  capitalize  on  the  unique  characteristics  of  ITSs: 

•  The  ability  to  generate  instruction,  rather  than  require  developers  to  foresee  and 
pre-store  all  possible  interactions  between  users  and  technology 

•  The  ability  to  support  mixed  initiative  dialogue,  in  which  either  the  user  or  the 
technology  can  initiate  open-ended  discussion. 

The  ADL  initiative  and  the  development  of  ITSs,  then,  have  several  key  goals  in 
common: 

•  Both  are  generative  since  they  envision  the  generation  of  presentations  on 
demand,  in  real  time. 

•  Both  are  intended  to  provide  presentations  that  are  tailored  in  content,  sequence, 
level  of  difficulty,  level  of  abstraction,  style,  and  so  forth  to  users’  intentions, 
backgrounds,  and  needs. 

•  Both  have  a  stake  in  research  intended  to  accomplish  such  individualization. 
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•  Both  can  be  used  equally  well  to  aid  learning  or  decision-making. 

•  Both  are  intended  to  accommodate  mixed  initiative  dialogue,  in  which  either  the 
technology  or  the  user  can  initiate  or  respond  to  open-ended  inquiries  or  dis¬ 
cussion. 

•  Both  will  benefit  greatly  from  a  supply  of  sharable  instructional  objects  readily 
available  for  the  generation  of  instructional  (or  decision-aiding)  presentations. 

Accordingly,  ADL  is  focused  on  the  specification  of  instructional  objects  that  meet 
several  criteria.  Among  these  criteria,  four  seem  most  relevant: 

1.  Accessible.  It  must  be  possible  to  find  needed  and  sharable  objects,  and  they 
must  be  accessible.  Basically,  we  need  widely  accepted  and  standard  ways  to 
store  objects  so  that  widely  accepted  and  standard  ways  can  be  used  to  find  and 
retrieve  them. 

2.  Interoperable.  Once  found,  the  objects  should  be  usable.  This  means  that  they 
must  be  interoperable  and  portable  across  most — if  not  all — ^platforms, 
operating  systems,  browsers,  and  courseware  tools. 

3.  Durable.  Once  implemented,  the  objects  should  continue  to  operate  reliably. 
They  should  be  durable.  If  the  underlying  platform,  operating  system,  or 
browser  is  modified  (e.g.,  when  a  new  version  is  released  and  installed), 
courseware  objects  should  continue  to  operate  as  before. 

4.  Reusable.  Finally,  courseware  objects  should  be  reusable.  Other  platforms, 
operating  systems,  browsers,  and  courseware  tools  should  be  able  to  reuse,  and 
perhaps  even  modify  as  needed,  the  original  courseware  objects. 

Specifications  intended  to  meet  these  criteria  are  being  developed  through  an 
evolving  series  of  specifications  called  the  Sharable  Content  Objects  Reference  Model 
(SCORM).  Versions  of  SCORM  can  be  found  at  the  ADL  web  site  (www.adlnet.org). 

We  expect  sharable  instructional  objects  to  reduce  the  costs  and  increase  the 
effectiveness  of  ADL  technologies  and  ITSs.  Development  of  these  objects  should  be 
cooperatively  promoted  and  pursued  by  the  ADL  and  ITS  communities.  Doing  so  will  allow 
high-quality  instruction  and  decision-aiding  to  become  ubiquitous  and  affordable. 
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WHAT  DO  SHARABLE  INSTRUCTIONAL  OBJECTS 
HAVE  TO  DO  WITH  INTELLIGENT  TUTORING  SYSTEMS,  AND 

VICE  VERSA? 


The  techniques  of  object-oriented  programming  are  influencing  many  areas  of  soft¬ 
ware  development.  Among  these  areas  are  the  development  of  tools,  materials,  and  applica¬ 
tions  for  technology-based  education  and  training.  This  document  discusses  the  value  of 
technology-based  instruction  in  general,  intelligent  tutoring  systems  (ITSs)  in  particular,  and 
how  the  specification  and  development  of  sharable  instructional  objects  can  enhance  both. 

A.  WHAT  ARE  THE  CONTRIBUTIONS  OE  TECHNOLOGY-BASED 
INSTRUCTION? 

Substantial  improvements  in  instructional  effectiveness  can  be  obtained  by  tailoring 
instruction  to  the  needs  and  capabilities  of  individual  learners.  Studies  of  one-on-one 
tutoring  provide  evidence  for  these  improvements.  One  widely  cited  discussion  was  based 
on  studies  performed  by  Benjamin  Bloom.  These  studies  compared  the  achievement  of 
individually  tutored  students  (one  instructor  for  each  student)  with  that  of  classroom  stu¬ 
dents  (one  instructor  for  every  28-32  students)  (Bloom,  1984).  It  is  not  surprising  to  find 
that  individual  tutoring  in  these  studies  increased  the  achievement  of  students.  What  is  sur¬ 
prising  is  the  magnitude  of  the  increase.  Bloom  reported  that  the  overall  difference  in 
achievement  across  three  studies  was  about  two  standard  deviations,  which  means,  roughly, 
that  tutoring  improved  the  achievement  of  50th  percentile  students  to  that  of  98th  percentile 
students.  Two  standard  deviations  is  a  large  difference.  Bloom  posed  it  as  a  challenge  to 
researchers  and  developers  in  education. 

Why  is  this  2-Sigma  difference  (as  Bloom  called  it)  a  challenge?  Why  do  we  not 
simply  provide  one-on-one  tutoring  for  all  our  students?  The  answer  is  straightforward  and 
obvious:  we  cannot  afford  it.  Providing  one  instructor  for  each  student  is,  in  most  cases, 
prohibitively  expensive.  Because  of  this  problem,  Michael  Scriven  early  on  described  indi- 
viduahzed,  tutorial  instruction  as  both  an  instructional  imperative  and  an  economic  impos¬ 
sibility  (Scriven,  1975). 
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We  may  now  have  the  means  to  break  out  of  this  dilemma.  Fundamental  to  the  value 
of  our  rapidly  evolving  computer  technology  is  its  ability  to  adjust  its  actions  in  real  time 
and  on  demand  to  the  conditions  of  the  moment.  The  promise  of  this  capability  for 
instruction  has  not  been  lost  on  researchers  and  developers.  Tutorial  instruction  and  tech¬ 
nology-based  instruction  use  this  ability  to  adjust  actions  responsively  to  meet  the  needs  of 
individual  students  to  solve  two  significant  problems:  individualized  pace  and  interactivity. 
These  problems  confront  all  classroom  teachers  and  may  account  for  much  of  the  difference 
in  effectiveness  between  individualized,  tutorial  instruction  and  classroom  instruction. 

1.  The  Challenges  of  Classroom  Instruction:  Pace 

The  variability  in  time  that  different  students  take  to  learn  and  that  teachers  must 
accommodate  in  classrooms  can  be  overwhelming.  Despite  conscientious  efforts  to  sustain 
common  levels  of  prior  knowledge  in  classrooms,  our  current  school  practices  appear  to 
increase  these  differences  by  about  one  year  for  every  year  students  spend  in  elementary 
school  (Heuston,  1997).  For  instance,  the  average  spread  of  academic  achievement  in  grade 
three  is  about  3  years.  By  grade  six,  it  increases  to  about  6  years. 

Research  has  helped  confirm  the  extent  of  this  variability.  Early  on,  Suppes  (1964) 
found  that  the  time  needed  by  Kindergarten  students  to  learn  to  build  words  from  letters 
varied  by  about  13:1.  Ratios  for  grade  5  students  to  master  a  unit  of  social  studies  were 
found  to  be  3:1  and  5:1  (Gettinger  and  White,  1980).  In  two  different  studies,  the  rates  of 
learning  observed  among  hearing-impaired  and  Native  American  students  were  found  to  be 
4:1  (Suppes,  Fletcher,  andZanotti,  1975  and  1976).  Based  on  a  range  of  research  findings, 
Carroll  (1970)  estimated  the  overall  ratio  of  5:1  for  elementary  school  students.  Even  among 
highly  selected  students  at  a  major  research  university,  the  times  needed  by  undergraduates 
to  learn  a  programming  language  have  varied  by  7:1  (Corbett,  1998,  personal  commu¬ 
nication).  The  latter  result  helps  confirm  earlier  observations:  even  though  rates  of  learning 
may  be  determined  initially  by  basic  ability,  the  effects  of  ability  are  quickly  overtaken  by 
those  of  prior  knowledge  and  experience  with  the  subject  matter  (Tobias,  1989). 

What  happens  when  we  allow  students  to  progress  at  their  own  rates,  when  we 
individualize  the  pace  of  learning  as  tutors  do  and  as  we  have  long  been  able  to  do  using 
technology-based  instruction?  Some  studies  have  compared  the  time  to  achieve  specific 
instructional  objectives  in  classrooms  with  the  time  to  achieve  the  same  objectives  using 
technology-based  instruction.  Reviews  combining  the  results  of  these  studies  have  been 
reported.  Orlansky  and  String  (1979)  found  that  reductions  in  time  to  reach  instructional 
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objectives  averaged  about  54  percent  in  their  review  of  computer-based  instruction  (CBI) 
used  in  military  training.  Fletcher  (1997)  reported  an  average  time  reduction  of  31  percent  in 
6  studies  of  interactive  multimedia  instruction  applied  in  higher  education.  Kulik  and  his 
colleagues  found  time  reductions  of  34  percent  in  17  studies  of  CBI  used  in  higher  educa¬ 
tion  and  24  percent  in  15  studies  of  adult  education  (Kulik,  1994).  AH  these  reviews  are 
effectively  independent  since  they  reviewed  different  sets  of  evaluation  studies.  On  this 
basis,  reductions  of  about  30  percent  in  the  time  students  take  to  reach  a  variety  of  given 
instructional  objectives  seem  to  be  a  reasonable  expectation. 

The  individualization  of  pace  allowed  by  technology-based  instruction,  including 
ITSs,  is  likely  to  produce  significant  savings  in  time  to  learn.  Do  these  savings  in  time 
matter?  Studies  have  shown  that  the  Department  of  Defense  (DoD)  might  save  hundreds  of 
millions  of  dollars  in  specialized  skill  training  if  the  training  time  to  reach  specified  thresh¬ 
olds  of  knowledge  and  skill  can  be  reduced  by  30  percent  (National  Research  Council, 
1997).  These  savings  result  as  much  from  reductions  in  training  costs  as  from  the  value  of 
providing  trained  personnel  ready  for  duty  in  operational  units  earlier. 

We  might  expect  such  savings  to  be  substantial  and  operationally  significant  in 
military  and  industrial  training,  where  the  organization  providing  the  training  pays  the  stu¬ 
dents  and  where  the  employers  and  the  students  have  a  stake  in  the  rapid  mastery  of  subject 
matter  and  the  early  availability  for  organizational  operations.  However,  these  savings  can 
also  be  equally  valuable  in  K-12  education.  As  Gettinger  and  White  (1980)  asked,  why 
should  we  assume  that  time  to  learn  is  unimportant  to  school  children?  AH  societies  have  a 
heavy  stake  in  their  success.  Opportunities,  such  as  those  provided  by  technology-based 
instruction  (i.e.,  permitting  individual  students  to  expand  their  capabilities  and  realize  their 
potential  as  rapidly  as  possible),  may  be  even  more  important  than  they  are  in  military  and 
industrial  training. 

2.  The  Challenges  of  Classroom  Instruction:  Interactivity 

If  we  consider  interactivity  to  be  question-and-answer  sequences  occurring  between 
students  and  instructors,  could  accounting  for  the  differences  we  observe  between  one-on- 
many  classroom  instruction  and  one-on-one  tutorial  instruction  also  help?  Those  who  study 
classroom  interactions  of  this  sort  have  found  that  groups  of  students  ask  about  3  questions 
an  hour  and  that  any  single  student  in  a  class  asks  about  0.11  questions  an  hour  (Graesser 
and  Person,  1994).  By  contrast,  students  in  individual  tutorial  sessions  have  been  found  to 
ask  20-30  questions  an  hour  and  have  been  required  to  answer  1 17-146  questions  an  hour. 
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Finally,  some  students  taking  CBI  may  answer  8-12  questions  a  minute.  These  questions 
have  been  selected  to  meet  their  individual  needs  and  are  immediately  graded  to  provide 
instant  feedback. 

The  intensity  of  tutorial  instruction  provided  by  technology  evidently  pays  off  in 
student  achievement.  In  a  study  that  compared  97  empirical  evaluations  made  up  almost 
entirely  of  standard  (non-ITS)  CBI  with  conventional  classroom  approaches,  Kulik  (1994) 
reported  an  effect  size  advantage  for  computer-based  approaches  of  about  0.39  standard 
deviations,  or  roughly  an  improvement  of  50th  percentile  students  to  about  the  65th  per¬ 
centile  of  achievement.  In  an  attempt  to  determine  the  advantages  to  instruction  added  by 
multimedia  capabilities,  Fletcher  (1997)  summarized  a  review  of  47  comparisons  of  inter¬ 
active  multimedia  instruction  with  conventional  classroom  approaches  and  found  an  effect- 
size  advantage  for  these  technology-based  approaches  of  about  0.50  standard  deviations,  or 
roughly  an  improvement  of  50th  percentile  students  to  about  the  69th  percentile  of 
achievement.  Fletcher  also  reported  on  11  ITS  evaluations  and  found  an  effect  size  advan¬ 
tage  for  ITSs  of  about  0.84  standard  deviations,  roughly  an  improvement  of  50th  percentile 
students  to  about  the  79th  percentile  of  achievement.  In  a  review  of  5  evaluations  of  the 
SHERLOCK  ITS  system,  Gott,  Kane,  and  Lesgold  (1995)  report  an  overall  effect  size 
advantage  of  about  1.05  standard  deviations,  roughly  an  improvement  of  50th  percentile 
students  to  about  the  85th  percentile  of  achievement. 

We  cannot  say  if  we  will  achieve  Bloom’s  target  improvement  of  two  standard 
deviations,  but  the  available  evidence  suggests  that  we  are  progressing  in  the  right  direction 
through  the  use  of  technology-based  instruction.  These  findings  also  suggest  that  ITSs 
raise  the  ceiling  for  the  effectiveness  of  technology-based  instruction  and  provide  new  ave¬ 
nues  for  improvement  over  standard  classroom  instruction.  We  may  be  approaching  the 
limits  of  the  improvements  we  can  achieve  with  general,  non-ITS  CBI;  however,  we  also 
have  much  to  learn  about  the  instructional  possibilities  for  ITSs. 

3.  Are  We  Getting  Anywhere?:  Costs 

The  preceding  results  show  promise  for  technology-based  instruction  in  general  and 
ITSs  in  particular.  However,  if  we  want  to  make  a  difference  in  the  day-to-day  practice  of 
instruction,  we  must  address  the  concerns  of  those  who  are  making  decisions  about  it.  For 
them,  effectiveness  is  an  important  consideration  but  is  only  one  component  of  the  equation. 
The  hallmark  of  administrative  decision-making  is  the  consideration  of  what  must  be 
surrendered  to  gain  some  advantage.  Generally,  these  decisions  involve  tradeoffs  between 
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costs  and  effectiveness.  Decision-making  in  instruction  is  no  different.  We  must  consider 
costs  and  effectiveness. 

The  ratios  of  costs  for  technology-based  instruction  compared  with  more  conven¬ 
tional  instruction  can  be  reported  in  three  categories: 

1.  The  initial  investment  costs  to  develop  and  implement  both  types  of  training 
(technology-based  instruction  and  CBI) 

2.  The  operating  and  support  costs  for  both  types  of  training  once  they  are  in 
place 

3.  These  two  cost  categories  combined. 

In  reports  structured  for  this  purpose,  the  smaller  the  ratio,  the  better  the  news  for 
the  technology-based  training.  For  one  set  of  studies  in  which  achievement  under 
technology-based  training  was  at  least  equal  and  generally  superior  to  that  produced  by 
more  conventional  instruction,  Fletcher  (1997)  reported  the  ratios  to  be  0.43  for  initial 
investment,  0.16  for  operating  and  support,  and  0.35  overall.  In  these  studies,  substituting 
simulated  equipment  for  the  real  equipment  to  be  used  on  the  job  resulted  in  most  of  the 
cost  savings  achieved  by  technology-based  instruction.  Nonetheless,  these  ratios  suggest 
that  substantial  economies  can  be  realized  through  the  use  of  technology-based  instruction 
and  that  the  return  can  justify  the  investment.  More  complete  analyses,  based  on  projections, 
suggest  that  the  potential  savings — not  just  cost  avoidances — for  DoD  may  amount  to 
hundreds  of  millions  of  dollars  (National  Research  Council,  1997). 

4.  In  Summary:  The  Thirds 

Assessments  of  technology-based  instruction  leave  us  with  “the  thirds.”  Use  of 
technology  reduces  the  cost  of  instruction  by  about  one-third,  and  it  also  either  reduces  time 
to  reach  given  instructional  objectives  by  one-third  or  it  increases  the  achievement  of  its 
students  by  about  one-third.  The  primary  payoff  for  many  organizations  is,  of  course,  the 
rapid  preparation  of  personnel  to  perform  operational  duties. 

B.  WHAT  ARE  THE  CONTRIBUTIONS  OE  ITSs? 

It  may  be  best  to  begin  by  noting  the  features  that  garden-variety  CBI  can  and  does 
provide.  Notably  it  can: 

•  Accommodate  an  individual  student’s  rate  of  progress,  allowing  as  much  or  as 
little  time  that  an  individual  student  needs  to  reach  instructional  objectives 

•  Adjust  the  sequence  of  instructional  content  to  each  student’s  needs 
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•  Adjust  the  content  itself,  ensuring  that  different  students  receive  different  con¬ 
tent  depending  on  what  they  have  mastered  and  what  they  have  yet  to  learn 

•  Make  the  instruction  as  easy  or  as  difficult  as  necessary 

•  Adjust  to  the  learning  style  (e.g.,  verbal  vs.  visual)  that  is  most  appropriate  for 
each  student. 

These  capabilities  have  been  available  and  used  in  CBI  since  its  inception  in  the  1950s  (e.g., 
Atkinson  and  Fletcher,  1972;  Suppes  and  Momingstar,  1972;  Fletcher  and  Rockway,  1986). 
Those  who  promote  systems  with  these  features  by  touting  them  as  indicators  of  newly 
developed  “intelligent”  capabilities  may  be  missing  some  history.  Whatever  the  case,  they 
are  using  the  term  “intelligent”  in  ways  that  differ  from  the  historical  objectives  of 
ITSs — objectives  that  have  been  pursued  since  the  late  1960s. 

What  are  these  objectives?  What  is  left  for  ITSs  to  provide?  What  can  we  get  from 
ITSs  that  is  not  otherwise  available?  Two  functionalities  deserve  mention: 

1 .  The  ability  to  allow  either  the  computer  or  the  student  to  ask  open-ended  ques¬ 
tions  and  initiate  instructional,  “mixed-initiative”  dialogue  as  needed  or  desired 

2.  Related  to  the  preceding  functionality,  the  ability  to  generate  instructional  mate¬ 
rial  and  interactions  on  demand  rather  than  require  developers  to  foresee  and 
pre-store  all  such  materials  and  interactions  needed  to  meet  all  possible  eventu- 
ahties. 

The  first  of  these  functionalities  requires  the  ITS  to  understand  and  participate  in 
mixed-initiative  interactions  with  the  student.  It  requires  a  mutual  understanding  of  a  lan¬ 
guage  for  information  retrieval,  decision-aiding,  and  instruction  that  is  shared  by  the  ITS 
and  the  student/user.  Natural  language  has  been  a  frequent  choice  for  this  capabihty,  but  the 
language  of  mathematics,  mathematical  logic,  and  electronics  have  been  used,  as  reported  in 
publications  edited  by  Suppes  (1981),  Sleeman  and  Brown  (1982),  Psotka,  Massey,  and 
Mutter  (1988),  and  Farr  and  Psotka  (1992). 

Whatever  form  this  dialogue  takes,  the  mixed-initiative  dialogue  (in  which  either  the 
student  or  the  instructor  can  initiate  interactions)  appears  to  be  a  key  feature  of  one-on-one 
tutorial  instruction  (e.g.,  Graesser,  Person,  and  Magliano,  1995).  In  attempting  to  secure  for 
our  students  the  benefits  of  one-on-one  tutorials,  we  may  turn  to  technology-based 
instruction  that  specifically  seeks  to  develop  and  implement  mixed-initiative  dialogue  as  an 
instructional  approach.  Such  a  capability  has  been  a  goal  of  intelligent  tutoring  systems 
(Carbonell,  1970)  for  a  long  time. 
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The  second  functionality  requires  ITSs  to  devise  on  demand — not  retrieve  from 
storage — interactions  and  presentations  for  individual  students.  This  capability  involves 
more  than  generating  elements  to  fill  in  blanks  in  a  template.  It  means  generating  interac¬ 
tions  and  presentations  from  information  primitives  using  an  “instructional  grammar”  that 
is  analogous  to  the  deep-structure  grammar  of  the  transformational-generative  linguists  of  a 
generation  ago.  This  functionality  harkens  back  to  the  roots  of  ITS  development,  as  (again) 
can  be  seen  in  the  volumes  edited  by  Suppes  (1981),  Sleeman  and  Brown  (1982),  Psotka, 
Massey,  and  Mutter  (1988),  and  Farr  and  Psotka  (1992). 

Motivations  for  both  these  functionalities  can  be  found  in  basic  research  into  human 
learning,  memory,  perception,  and  cognition.  Findings  from  this  research  have  led  us  to  view 
all  cognitive  processes  as  constructive  and  regenerative.  They  have  caused  general  theories 
of  perception  and  learning  to  evolve  from  the  fairly  strict  and  logical  positivism  of 
behavioral  psychology  (which  emphasized  the  study  of  directly  observable  and  directly 
measurable  actions)  to  a  greater  consideration  of  internal,  less  observable  processes  that  are 
assumed  to  mediate  and  enable  human  learning  and  to  produce  the  directly  observable 
behavior  that  is  the  subject  of  behaviorist  approaches.  The  keynote  of  these  newer  concep¬ 
tions  of  cognition  may  have  been  struck  by  Ulric  Neisser  who  stated,  “The  central  assertion 
is  that  seeing,  hearing,  and  remembering  are  all  acts  of  construction,  which  may  make  more 
or  less  use  of  stimulus  information  depending  on  circumstances.”  (1967,  p.  10). 

Neisser  was  led  to  this  point  of  view  by  a  large  body  of  empirical  evidence  showing 
that  many  aspects  of  human  behavior,  such  as  seeing  and  hearing,  simply  could  not  be 
accounted  for  by  external  physical  cues  reaching  human  perceptors,  such  as  eyes  and  ears. 
Additional  processes,  including  an  internally,  one  might  say  cognitively,  generated  analysis 
by  synthesis,  had  to  be  posited  to  account  for  well-established  and  observable  human  abili¬ 
ties  to  detect,  identify,  and  process  physical  stimuli.  Such  a  process  requires  an  active 
synthesis  of  the  environment  based  on  a  runnable  cognitive  model,  or  simulation,  that  is 
vahdated  or  modified  as  needed  by  cues  being  received  from  sensory  perceptors.  It  is  the 
actively  evolving  simulation — not  the  stimuli  alone — that  is  said  to  account  for  what  the 
individual  learns. 

These  ideas  were,  of  course,  around  long  before  Neisser  published  his  book,  and  it 
is  notable  that  they  did  not  occur  only  to  psychologists.  Norbert  Weiner,  a  cyberneticist, 
suggested  that,  “In  both  [the  living  individual  and  the  machine],  there  exists  a  special  appa¬ 
ratus  for  collecting  information  from  the  outside  world  at  low  energy  levels  and  for  making 
it  available  for  the  operation  of  the  individual  or  of  the  machine.  In  both  cases,  these  external 
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messages  are  not  taken  neat,  but  through  the  internal  transforming  powers  of  the  apparatus, 
whether  it  be  alive  or  dead.  The  information  is  then  turned  into  a  new  form  available  for  the 
further  stages  of  performance”  (1954,  pp.  26-27;  italics  are  Neissers). 

All  this  leads  to  the  impression  that  the  generative  capability  sought  by  ITSs  and  by 
Advanced  Distributed  Learning  (ADL)  is  not  something  merely  nice  to  have  but  is  essential 
if  we  are  to  advance  beyond  the  constraints  of  the  prescribed  branching,  programmed 
learning,  and  the  ad-hoc  principles  currently  used  to  design  technology -based  instruction.  A 
generative  approach  is  essential  if  we  are  to  deal  successfully  with  the  immensity,  extent,  and 
variability  of  human  cognition. 

The  key  defining  characteristic  of  ITSs  and  ADL,  then,  is  not  the  application  of 
computer  techniques  from  artificial  intelhgence  (AI)  or  knowledge  representation  or  the 
specification  of  sharable  instructional  objects,  important  as  these  may  be.  Rather,  it  is  the 
functional  capability  to  generate  in  real  time  and  on  demand  instructional  interactions  that 
are  tailored  to  student  requests  and/or  needs.  This  generative  capability  motivated  DoD  to 
invest  in  ITSs  originally  (Fletcher  and  Rockway,  1986).  At  that  time,  the  motivation  was  to 
reduce  or  eliminate  the  high  costs  of  foreseeing  or  predicting  all  possibly  needed  materials 
and  interactions,  programming  them,  and  pre-storing  them  into  CBI.  Today,  this  motivation 
remains  fundamental  to  those  who  support  ITSs  and  ADL  development. 

1.  An  Example  of  an  ITS  at  Work 

An  example  of  these  capabilities  can  be  found  in  the  student-computer,  mixed- 
initiative  dialogues  supported  by  the  Sophisticated  Instructional  Environment  (SOPHIE),  an 
early  and  seminal  ITS  (Brown,  Burton,  and  DeKleer,  1982).  SOPHIE  used  a  general- 
purpose  electronic  simulator  to  represent  an  electronic  power  supply  and  create  a  reactive, 
tutorial  environment  for  training  in  electronic  troubleshooting.  The  underlying  simulator 
allowed  faults  introduced  into  the  system  by  a  (human)  instructor  or  by  SOPHIE  itself  to  be 
propagated  throughout  the  (simulated)  power  supply  in  a  realistic  fashion.  SOPHIE 
employed  a  semantic  network  of  information  about  the  power  supply — one  of  the  most 
sophisticated  natural  language  understanders  ever  built  into  an  ITS — and  an  articulate, 
expert  troubleshooting  capability  that  could  not  only  solve  arbitrary  faults  introduced  by 
students  or  instructors,  but  also  explain  to  the  student  how  it  solved  them. 

SOPHIE’S  reactive  environment  responded  in  a  realistic  fashion  to  the  actions  and 
decisions  of  a  student  engaged  in  practical  problem-solving.  It  employed  strategies  that 
shadowed  those  used  by  the  student  to  make  inferences  about  the  circuit  based  on  the 
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student’s  measurements.  SOPHIE  could  thereby  determine  which  deductions  the  student 
was  using,  construct  a  model  of  his/her  state  of  knowledge,  and  use  this  model  to  generate 
explanations  in  terms  that  the  student  was  prepared  to  understand.  SOPHIE  highlighted  the 
need  for  intelligent  training  systems  not  only  to  solve  arbitrary  problems,  but  to  do  so  in  a 
way  that  can  be  explained  to  students  and  be  understood  by  them.  It  pioneered  capabihties 
currently  lacking  in  many  information  acquisition  and  retrieval  systems. 

The  following  is  a  student-computer  dialogue  adapted  from  SOPHIE: 

Student:  What  is  the  base  emitter  voltage  of  the  voltage  limiting 
transistor? 

SOPHIE:  The  base  emitter  voltage  of  Q5  is  -  0.37  volts. 

The  first  student-SOPHIE  interaction  in  this  example  is  a  straightforward  question 
and  answer.  SOPHIE  has  introduced  a  fault  into  the  simulated  power  supply  and  the  stu¬ 
dent  is  trying  to  locate  and  repair  it. 

Student:  What  about  for  the  current  limiting  transistor? 

SOPHIE:  The  base  emitter  voltage  of  Q8  is  .349  volts. 

The  second  student-SOPHIE  interaction  is  a  little  less  straightforward  because,  in 
addition  to  simulating  the  faulted  power  supply,  SOPHIE  must  be  able  to  parse  the  dialogue 
sufficiently  to  know  that  the  student  is  asking  for  the  base  emitter  voltage  of  transistor  Q8. 
The  meaning  of  the  student’s  question  depends  heavily  on  the  context  established  by  his 
prior  question. 

Student:  What  should  it  be? 

SOPHIE:  In  a  working  circuit,  the  base  emitter  voltage  of  Q8  is  .593 
volts. 

The  third  student-SOPHIE  interaction  continues  its  dependence  on  SOPHIE’s 
understanding  of  context  and  also  requires  SOPHIE  to  extract  the  answer  by  simulating 
both  the  faulted  power  supply  and  one  that  is  operating  correctly. 

Student:  Replace  Q8. 

SOPHIE:  I  am  going  to  ask  you  some  questions  about  how  Q8  is 
faulted.  Are  any  junctions  shorted? 

The  fourth  student-SOPHIE  interaction  indicates  a  clear  step  beyond  what  Brown, 
Burton,  and  DeKleer  described  as  a  knowledgeable  system  to  what  they  considered  to  be  an 


9 


intelligent  system.  SOPHIE  has  progressed  from  a  knowledgeable  parsing  of  its  dialogue 
with  the  student  and  simulation  of  various  states  of  the  power  supply  to  a  system  exercising 
tutorial  intelligence.  It  shadowed  the  student’s  solution  path,  modeled  the  student’s 
troubleshooting  hypotheses,  determined  that  they  are  incorrect,  elected  to  capture  the 
dialogue  initiative  back  from  the  student,  and  undertook  a  series  of  tutorial  interactions 
intended  to  lead  the  student  to  a  more  correct  approach  to  the  problem. 

It  is  difficult  to  imagine  any  practical  way  to  achieve  this  level  of  instructional  func¬ 
tionality  without  the  generative  capability  and  mixed-initiative  dialogue  that  distinguishes 
ITSs  from  other  forms  of  CBI. 

2.  What  About  Decision-Aiding? 

As  can  be  seen,  an  ITS  such  as  SOPHIE  has  many  elements  and  capabilities  in 
common  with  a  decision-aiding  system,  intended  in  this  case  to  assist  an  electronics  mainte¬ 
nance  technician.  The  differences  may  lie  entirely  in  the  intentions  of  the  user  and  depend 
on  whether  the  system  is  being  used  to  solve  a  problem  or  to  effect  a  (more  or  less)  persis¬ 
tent  change  in  the  user’s  knowledge  and  skills.  The  underlying  computational  capabihties 
appear  to  be  the  same. 

Is  it  worth  expanding  the  capabilities  of  an  ITS  to  support  decision-aiding?  The 
U.S.  Air  Eorce’s  development  and  evaluation  of  the  Integrated  Maintenance  Information 
System  (IMIS)  provided  some  evidence.  Teitelbaum  and  Orlansky  (1996)  summarized  this 
evidence.  They  reviewed  and  analyzed  the  performance  of  technicians  who  were  specially 
trained  to  troubleshoot  and  maintain  E-16  avionics  systems  and  compared  the  performance 
of  these  technicians  with  the  performance  of  less  specifically  (and  much  less  expensively) 
trained  General  Aircraft  Technicians. 

Table  1  shows  the  results  of  this  comparison.  Notice  that  the  performance  of  the 
General  Aircraft  Technicians  became  essentially  the  same  as  that  of  the  Avionics  Specialists 
once  they  were  trained  on  the  IMIS.  Perhaps  more  notable,  however,  is  that  the  performance 
of  the  General  Aircraft  Technicians  using  IMIS  was  superior  to  that  of  Avionics  Specialists 
using  the  paper-based  Technical  Orders  currently  provided  as  maintenance  aids.  Einally,  it 
may  be  worth  noting  that  because  IMIS  was  tied  into  the  inventory  and  parts  databases,  it 
could  generate  the  paperwork  needed  to  order  new  parts  in  about  a  minute  for  both  groups. 
IMIS  provided  both  sophisticated  decision-aiding  to  the  technicians  as  well  as  help  in 
completing  clerical  tasks  of  a  less  exotic  nature. 
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Table  1.  Results  From  IMIS  Trials 


Correct 

Solutions 

Time  To 

Solve  Problems 

Time  To 

Order  Parts 

Technical 

Orders 

IMIS 

Technical 

Orders 

IMIS 

Technical 

Orders 

IMIS 

Avionics 

Specialists 

82% 

1 00% 

149  min 

124  min 

19  Min 

1  min 

General  Aircraft 
Technicians 

69% 

98% 

176  min 

124  min 

25  Min 

1  Min 

IMIS  supported  decision-aiding  only  for  three  subsystems  of  F-16  avionics.  Even  at 
that,  if  its  use  were  expanded  to  the  full  fleet  of  F-16s  for  these  three  subsystems,  net  cost 
savings  to  the  Air  Force  would  be  about  $23  million  annually.  Therefore,  using  ITS 
techniques  and  capabilities  to  provide  instruction  and  decision-aiding  appears  to  offer  sub¬ 
stantial  benefits. 

The  emphasis  on  IMIS  as  a  decision-aiding  system,  rather  than  just  a  maintenance- 
aiding  system,  is  intended  to  suggest  that  this  approach  is  not  limited  to  maintenance  or 
operation  of  devices.  Its  capabilities  could  be  applied  to  more  abstract  undertakings,  such  as 
command  and  control  (C2)  in  the  military  and  management  and  administration  in  the  civUian 
sector. 

3.  What  is  the  Underlying  Technical  Idea? 

Achieving  these  instructional  and  decision-aiding  functionalities  is  an  important 
goal,  but  it  requires  a  technical  idea  to  make  it  feasible.  Jaime  Carbonell  (1970)  first  articu¬ 
lated  this  idea.  He  recommended  replacing  the  ad-hoc  frame-oriented  (AFO)  instmctional 
approaches,  such  as  those  seen  in  programmed  texts  and  best  described  as  “intrinsic  pro¬ 
gramming”  (Crowder,  1959),  with  an  information-systems-orientated  (ISO)  approach.  The 
AFO  approach  of  intrinsic  programming  is  widely  used  in  technology-based  instruction.  It 
features  a  presentation  (often  a  paragraph  of  text  and/or  a  graphic  of  some  sort)  followed  by 
questions.  A  student’s  answers  to  these  questions  lead  to  different  responses,  usually  dif¬ 
ferent  branching  paths,  by  the  system.  The  presentations,  the  questions,  and  aU  possible 
branching  paths  must  be  anticipated  and  pre-specified  by  the  instruction  developers. 

CarbonelTs  ISO  approach — or,  as  we  might  say  today,  an  approach  based  on 
knowledge  representation — allowed  the  instructional  system  to  be  generative.  As  suggested 
some  time  ago  (Fletcher,  1975),  the  ISO  functionalities  possessed  by  SOPHIE  and  other 
ITSs  involve  computer  representation  or  modeling  of  the  student,  the  subject  matter,  and 
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expert  tutoring.  This  point  of  view  now  seems  to  be  commonly  accepted  and  is  found  to  a 
significant  degree  in  the  ITS  architectures.  Carbonell  (1970)  and  Woolf  and  Regian  (2000), 
among  many  others,  have  emphasized  knowledge  representation  as  a  key  underlying  capa¬ 
bility  for  ITSs.  Progress  is  being  made.  Woolf  and  Regian  emphasize  that  our  ability  to 
represent  human  cognition  has  gained  considerable  potency  because  of  the  advances  made 
in  cognitive  science  during  the  last  10  years.  These  advances  should  substantially  improve 
the  promise  and  capabilities  of  ITSs. 

C.  WHERE  IS  THIS  TAKING  US? 

1.  The  Third  Revolution  in  Learning 

The  emphasis  here  on  instructional  technology  brings  us  to  revolutions  in  instruc¬ 
tion.  The  first  of  these  may  have  occurred  with  the  development  of  written  language  about 
7,000  years  ago.  It  allowed  the  content  of  advanced  ideas  and  teaching  to  transcend  time  and 
place.  The  second  revolution  in  instruction  began  with  the  technology  of  books.  Books 
made  the  content  of  high-quality  instruction  again  available  anywhere  and  anytime  but  also 
inexpensive  and  accessible  to  many  more  people.  A  third  revolution  in  instruction  appears  to 
be  accompanying  the  introduction  of  computer  technology.  The  capabihty  of  this  tech¬ 
nology  for  real-time  adjustment  of  instructional  content,  sequence,  scope,  difficulty,  and 
style  to  meet  the  needs  of  individuals  suggests  a  third  pervasive  and  significant  revolution  in 
instruction.  It  makes  both  the  content  and  the  interactions  of  high-quality  instruction  widely 
and  inexpensively  accessible — again  anytime,  anywhere. 

2.  The  ADL  Initiative 

Building  on  this  possibility,  the  ADL  initiative  is  now  being  undertaken  in  the 
United  States  and  elsewhere.  This  initiative  is  intended  to  provide  education,  training,  and 
decision-aiding  (or  “mentoring”)  anytime,  anywhere.  It  capitalizes  on  the  growth  of  elec¬ 
tronic  commerce  and  the  World  Wide  Web.  It  takes  advantage  of  this  global — almost  irre¬ 
sistible — activity,  accelerates  it,  and  applies  it  to  learning.  It  will  help  ensure  the  availabihty 
of  human  competence  in  many  organizations  as  they  face  the  challenges  of  the  2L‘  century. 

The  ADL  initiative  has  been  tasked  to  provide  guidelines  and  specifications  for  all 
instructional  activities  of  the  United  States  Federal  Government.  It  is  similarly  expected  to 
provide  guidelines  and  specifications  for  the  instructional  activities  of  many  countries  in 
Europe  and  in  the  Pacific  Rim.  It  may  succeed  because  it  is  not  imposing  standards  devel¬ 
oped  by  the  government,  but,  rather,  is  pulling  together  the  best  ideas,  guidelines,  and 
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specifications  developed  by  a  variety  of  industrial,  academic,  and  government  sources  in  the 
United  States  and  elsewhere.  It  has  become  a  massive  cooperative  development  that  involves 
economic  sectors  focused  on  achieving  the  ADL  initiative’s  goals,  which  are  to  provide 
learning  and  decision-aiding  capabilities  tailored  to  the  needs  of  individuals  anywhere  and 
anytime  they  are  needed. 

D.  WHAT  DOES  ADL  HAVE  TO  DO  WITH  ITSs? 

1.  ADL  and  Intelligent  Instructional  Systems 

In  the  context  of  ITSs,  “intelligent”  refers  as  much  to  our  intentions  as  our  results; 
however,  it  is  more  than  a  marketing  term.  It  refers  to  the  specific  functionalities  that  are  the 
goals  of  ITS  development.  As  discussed,  these  functionalities  are  distinct  from  those  found 
in  more  conventional  approaches  to  CBI.  They  require  ITSs  to  generate  instruction  in  real 
time  and  on  demand  as  required  by  individual  learners.  This  generative  approach  is  also  the 
goal  of  the  ADL  initiative,  which  is  combining  the  benefits  of  both  object-oriented 
development  and  Web  delivery  with  those  of  technology-based  instruction  to  achieve  its 
objectives. 

Despite  common  goals,  the  ADL  initiative  and  the  development  of  intelligent 
instructional  systems  have  been  pursued  in  parallel  but  separately.  The  ADL  initiative  has 
focused  on  the  specification  of  sharable  instructional  objects  that  can  be  retrieved  from  and 
then  delivered  over  the  World  Wide  Web  anywhere,  anytime.  They  can  be  used  and  reused 
to  tailor  instruction  and  decision-aiding  presentations  to  the  needs  of  individual  learners. 
They  support  the  core  functional  capability  targeted  by  the  ADL  initiative:  making  the 
benefits  of  tutorial  instruction  and  decision-aiding  universally  and  affordably  available.  The 
common  interests  shared  by  the  developers  of  ITSs  and  those  pursuing  the  ADL  initiative 
have  been  noted  on  both  sides. 

The  ADL  initiative  is  guided  by  a  vision,  roughly  illustrated  in  Figure  1,  of  a  future 
in  which  everyone  has  an  electronic  personal  learning  associate.  This  device  will  assemble 
learning  or  decision-aiding  presentations  on  demand  and  in  real  time — anytime,  anywhere. 

The  presentations  will  be  exactly  tailored  to  the  needs,  capabihties,  intentions,  and 
learning  state  of  each  individual  or  group  (e.g.,  crew,  team,  or  staff)  of  individuals.  Com¬ 
munication  with  the  personal  learning  associate  will  be  based  on  natural  language  dialogue 
initiated  either  by  the  device  or  by  its  users.  The  device  will  be  small  enough  to  be  carried  in 
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Shareable  courseware  Assembled  in  To  provide  iearning 

objects  from  across  the  reai-time,  on-  and  assistance 

Worid  Wide  Web  demand  anytime,  anywhere 

Figure  1.  An  ADL  Future 

a  shirt  pocket,  or  it  will  be  wearable.  It  will  be  used  by  individuals  learning  by  themselves,  in 
groups,  or  in  classrooms.  It  will,  of  course,  be  wireless. 

Most  of  the  technology  needed  to  build  such  a  device  exists  now.  Although  we 
cannot  yet  fit  it  into  a  shirt  pocket,  advances  in  electronics  should  take  care  of  that.  What  is 
especially  needed  for  instruction  and  decision-aiding  is  content  in  the  form  of  instmctional 
objects,  which  we  are  calling  sharable  instructional  objects.  These  objects,  shown  in  the 
cloud  on  the  left  side  of  Figure  1,  must  be  readily  accessible  across  the  World  Wide  Web 
or  whatever  form  our  global  information  network  takes  in  the  future. 

Once  these  objects  exist,  they  must  be  identified,  selected,  and  assembled  in  real  time 
and  on  demand  and  then  handed  to  the  personal  learning  associates,  which  provide  the 
instruction  or  decision-aiding.  This  work  of  identifying,  selecting,  and  assembling  objects  is 
the  job  of  the  server,  represented  as  a  box  in  the  middle  of  Figure  1.  By  importing  “logic” 
or  instructional  strategy  objects,  the  server  [which  today  would  be  called  a  Learning 
Management  System  (LMS)]  can  acquire  the  capabilities  of  the  intelligent  tutoring/ 
decision-aiding  system  we  have  been  discussing.  If  it  does  so  successfully,  it  will  also 
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provide  the  benefits  we  have  been  discussing.  However,  in  the  ADL  initiative,  the  underlying 
processes  will  rely  on  sharable  instructional  objects. 

The  ADL  initiative  and  the  development  of  ITSs  have  several  key  goals  in  common: 

•  Both  are  generative  since  they  envision  the  generation  of  presentations  on 
demand,  in  real  time. 

•  Both  are  intended  to  provide  presentations  tailored  in  content,  sequence,  level  of 
difficulty,  level  of  abstraction,  style,  and  so  forth  to  users’  intentions,  back¬ 
grounds,  and  needs. 

•  Both  have  a  stake  in  the  research  intended  to  accomplish  such  individualization. 

•  Both  can  be  used  equally  well  to  aid  learning  or  decision-making. 

•  Both  are  intended  to  accommodate  mixed  initiative  dialogue  in  which  either  the 
technology  or  the  user  can  initiate  or  respond  to  inquiries  in  natural  language. 

•  Both  will  benefit  greatly  from  a  supply  of  sharable  instructional  objects  readily 
available  for  the  generation  of  instructional  (or  decision-aiding)  presentations. 

2.  Sharable  Instructional  Objects 

Roschelle  and  Kaput  (1995)  emphasized  the  promise  of  object-based  software,  such 
as  that  promoted  by  ADL,  for  combining  many  kinds  of  interactive  content  in  multiple 
display  formats  and  attaining  for  education  the  benefits  now  being  realized  in  business  from 
the  use  of  integrated  office  software.  Roschelle  et  al.  (1999)  illustrated  these  points  by 
examining  the  software  techniques  underlying  five  object-based  education  projects  that  have 
already  been  developed.  Gibbons  and  Fairweather  (2000)  weigh  these  issues  at  length  in 
examining  the  present  status  and  future  of  CBI. 

The  cloud  on  the  left  side  of  Figure  1  represents  the  World  Wide  Web  or  whatever 
we  will  use  in  the  future  to  provide  our  global  communication  ether.  One  crucial  matter  for 
the  implementation  of  ADL  is  what  has  loosely  been  called  “content”  in  the  form  of 
sharable  objects  represented  by  the  various  icons  shown  in  the  cloud.  People  involved  in 
ADL  have  spent — and  continue  to  spend — much  time,  effort,  and  energy  discussing  these 
sharable  content  objects  (SCOs).  This  matter  transcends  the  immediate  issues  of  ADL  as 
evidenced  by  discussions  edited  by  Wiley  (2000)  and  others. 

As  presently  defined,  SCOs  could  be  entire  courses,  lessons  within  courses,  or 
modules  within  lessons.  They  could  be  electronic  representations  of  media,  text,  images, 
sound,  Web  pages,  or  other  data  that  can  be  presented  to  students.  The  size,  or 
“granularity,”  of  SCOs  is  a  matter  of  considerable  discussion.  Gibbons,  Nelson,  and 
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Richards  (2000)  emphasized  that  SCOs  will  be  most  useful  if  they  are  prepared  in 
sufficiently  small  chunks  that  can  be  accessed  and  reused  by  other  instructional  materials. 
Access  must  be  rapid  and  easily  accomplished  across  whatever  form  our  future  global 
information  network  takes.  As  suggested  by  Figure  1,  once  these  SCOs  exist,  they  must  be 
available  for  assembly  in  real  time  and  on  demand  by  some  sort  of  servers  (middle  of 
Figure  1)  and  then  handed  to  client  personal  learning  associates  (right  side  of  Figure  1). 

SCOs  could  also  be  material  that  is  not  seen  by  students  but  is  needed  to  register 
them  for  courses,  report  on  their  progress,  collect  them  into  classes  and  other  administrative 
groupings,  or  store  data  needed  to  tailor  instruction  to  individual  student  needs.  Sig¬ 
nificantly,  they  could  also  be  content  in  the  form  of  algorithms  that  aggregate,  integrate,  and 
sequence  other  objects  as  needed  to  manage  the  progress  of  students  toward  their  attainment 
of  specific  instructional  outcomes. 

Some  researchers  have  provided  examples  and  suggested  some  architectures  for  this 
possibility.  Koedinger,  Suthers,  and  Forbus  (1999),  in  their  discussion  of  tutor  agents, 
emphasize  their  value  for  “higher-order”  reasoning,  such  as  that  associated  with  developing 
and  assessing  experimental  strategies,  determining  representation  strategies,  developing 
conjectures,  drawing  conclusions,  and  making  arguments  in  a  science  learning  space.  Ritter 
and  Koedinger  (1996)  suggest  a  general  architecture  that  translates  the  mouse  clicks,  key 
presses,  and  other  responses  of  students  into  semantic  descriptions  that  can  be  used  for 
tutoring  and  managing  student  progress  toward  curriculum  goals. 

These  activities  are  the  kinds  of  functions  that  LMSs  are  now  expected  to  perform. 
These  LMSs  are  intended  to  be  generalized  capabilities  for  many  different  types  of  students, 
subject  matters,  curriculum  structures,  instructional  approaches,  and  curriculum  objectives. 
However,  there  seems  to  be  a  clear  tradeoff  between  the  amount  of  functionality  that  can  be 
built  into  these  LMSs  and  their  generality.  The  more  functionality  built  into  them,  the  less 
general  purpose  they  become.  LMSs  that  import  these  functionalities  as  objects  are  likely  to 
be  more  flexible,  powerful,  and  generalizable. 

In  this  sense,  the  current  discussion  of  “content”  and  what  SCOs  might  be  echoes 
the  controversy  that  occurred  early  in  the  development  of  digital  computers:  whether  data 
(traditional  content)  and  logic  (algorithms)  should  be  stored  in  different  ways  and  in 
different  locations  of  the  early  machines  (Goldstine,  1972).  John  von  Neumann  settled  that 
discussion.  He  recommended  storing  data  and  logic  together  as  digital  bits  in  a  common 
memory.  The  nature  of  SCOs,  their  disposition,  and  whether  to  apply  a  solution  analogous 
to  von  Neumann’s  remain  to  be  determined. 
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Still,  SCOs  have  become  the  foundation  for  ADL.  The  availabihty  of  SCOs  will 
significantly  reduce  the  costs  of  preparing  instruction,  decision-aiding,  and  job-performance 
assistance  for  technology-based  dehvery.  This  is  likely  to  be  true  whether  the  SCOs  are 
assembled  in  advance  by  course  authors  and  developers  or,  as  suggested  by  Figure  1,  are 
assembled  on  demand  and  in  real  time  by  server  algorithms  incorporated  in  or  imported  into 
LMSs.  For  these  reasons,  ADL  development  is  presently  focused  on  packaging  SCOs  in 
anticipation  of  what  is  called  by  Spohrer,  Sumner,  and  Shum  (1998),  among  others,  the 
“educational  object  economy.”  The  primary  idea  behind  such  an  economy  is  that  the 
emphasis  in  preparing  materials  for  technology-based  instruction  (or  decision-aiding)  will 
shift  from  the  current  concern  of  preparing  content  components,  or  instructional  objects,  to 
one  of  integrating  already  available  content  into  meaningful  and  relevant  presentations. 

ADL  developers  recognize  that  this  software  engineering  concern  is  only  a  begin¬ 
ning.  The  primary  goal  of  ADL  is  not  to  promote  tinkering  with  software  objects,  but  to 
develop  the  functional  capability  of  producing  instructional  outcomes  anytime,  anywhere 
they  are  needed  or  desired.  The  ADL  initiative  has  made  substantial  progress  at  this  soft¬ 
ware  engineering  level,  but  it  must  also  address  real  learning  issues.  It  must  determine  how 
to  assemble  instructional  objects  to  achieve  targeted  instructional  objectives. 

These  two  areas  of  software  engineering  and  instructional  design  are  not  inde¬ 
pendent.  They  must  coordinated  and  “harmonized”  to  achieve  the  ADL  vision.  This  point 
suggests  that  designers  and  developers  need  some  understanding  of  the  underlying  ADL 
architecture  and  its  software  specifications  to  ensure  that  ADL  presentations  bring  about 
efficient  and  effective  learning.  It  also  suggests  that  the  software  engineers  designing  SCOs 
must  (along  with  their  meta-data  packaging  and  assumptions  concerning  course  structures, 
LMS  structures,  and  communication  protocols)  understand  and  accommodate  all  varieties  of 
instructional  approaches  and  what  each  requires  for  software  support. 

This  vision,  which  foresees  the  development  of  an  instructional  object  economy, 
must  start  with  the  specification  of  sharable  instructional  objects.  Fortunately,  many 
individuals  (technicians,  software  engineers,  instruction  designers,  cognitive  researchers,  and 
so  forth)  from  organizations  representing  all  sectors  of  the  economy  (government,  industry, 
and  academic)  in  the  United  States  and  elsewhere  in  Europe  and  Asia  have  joined  in  this 
quest.  Development  of  these  objects  has  become  a  global  effort.  Those  involved  in  the  U.S. 
ADL  initiative  mostly  need  to  orchestrate  the  effort  and  document  its  results. 

Sharable  instructional  objects  that  support  ADL  functional  requirements  must  meet 
some  criteria.  Among  these  criteria,  four  seem  most  prominent: 
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1.  Accessible.  It  must  be  possible  to  find  needed  and  sharable  objects,  and  they 
must  be  accessible.  Basically,  we  need  widely  accepted  and  standard  ways  to 
store  objects  so  that  widely  accepted  and  standard  ways  can  be  used  to  find  and 
retrieve  them. 

2.  Interoperable.  Once  found,  the  objects  should  be  usable.  This  means  that  they 
must  be  interoperable  and  portable  across  most — if  not  all — ^platforms, 
operating  systems,  browsers,  and  courseware  tools. 

3.  Durable.  The  objects  should  be  durable.  Once  implemented,  the  objects 
should  continue  to  operate  reliably.  If  the  underlying  platform,  operating 
system,  or  browser  is  modified  (e.g.,  when  a  new  version  is  released  and 
installed),  courseware  objects  should  continue  to  operate  as  before. 

4.  Reusable.  Finally,  courseware  objects  should  be  reusable.  Other  platforms, 
operating  systems,  browsers,  and  courseware  tools  should  be  able  to  reuse,  and 
perhaps  even  modify  as  needed,  the  original  courseware  objects. 

Specifications  intended  to  meet  these  criteria  are  being  developed  through  an 
evolving  series  of  specifications  called  the  Sharable  Content  Objects  Reference  Model 
(SCORM).  Versions  of  SCORM  can  be  found  at  the  ADL  website  (www.adlnet.org). 

Specifying  these  objects  arises  from  the  intersection  of  disciplines  concerned  with 
learning  and  cognition,  information  storage  and  retrieval,  and  software  engineering. 
Packaging  the  objects  so  that  they  are  accessible  will  depend  on  the  model  of  cognition  and 
learning  or  the  learning  strategy  trying  to  find  them.  This  packaging  requires  the  develop¬ 
ment  of  a  taxonomy  that  is  sufficiently  robust  to  accommodate  many  models  and  strategies. 
Aside  from  such  models  of  human  cognition,  learning,  and  instruction,  some  representation 
of  the  physical  world  is  also  required  to  accommodate  comprehensive  range  of  objects. 
Finally,  interoperability  and  reusability  require  techniques  of  software  engineering  that  are 
compatible  with  the  prevailing  global  ether — which  today  is  the  World  Wide  Web — and 
with  the  many  operating  environments  in  which  the  instructional  objects  will  be  used.  Given 
the  difficulty  of  the  undertaking,  it  is  fortunate  that  it  has  become  a  global  quest.  Current 
progress  toward  these  ends  can  be  viewed  through  many  portals.  The  ADL  website  is  a 
good  a  place  to  start. 

E.  SUMMARY 

This  discussion  suggests  that: 

•  Substantial  improvements  in  instructional  effectiveness  can  be  obtained  by 
tailoring  instruction  to  the  needs  and  capabilities  of  individual  learners.  Evi¬ 
dence  for  these  improvements  can  be  found  in  studies  of  one-on-one  tutoring. 
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However,  in  most  applications,  providing  one  instructor  for  each  student  is 
prohibitively  expensive. 

•  Using  technology  can  make  the  benefits  of  individualized  instruction  afford¬ 
able.  Evaluations  of  CBI  in  general  and  ITSs  in  particular  provide  evidence  of 
these  benefits. 

•  Combined  with  the  availability  of  sharable  instructional  objects  available  from 
the  Web,  ITSs  will  lead  to  the  development  and  wide  use  of  personal  learning 
associates,  which  will  allow  high-quality  instruction  and  decision-aiding  to 
become  ubiquitous  and  affordable. 

•  This  is  a  desirable  result. 

If  Kurzweil’s  (1999)  projections  are  correct,  computers  may  become  more  effective 
in  providing  instruction  than  human  tutors  even  if  humans  use  all  the  techniques  Graesser 
and  his  colleagues  found  they  now  neglect  (Graesser,  Person,  and  Magliano,  1995).  We 
may  not  be  implanting  integrated  circuits  in  our  brains  as  Kurzweil  suggests,  but  the  goal  of 
using  technology  to  discover  more  than  any  human  agent  can  about  the  unique  potential  of 
every  individual  and  devising  effective  and  individualized  procedures  to  reach  it  seems  an 
appealing  and  realistic  prospect. 
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computer-based  instruction 
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Institute  for  Defense  Analyses 

IMIS 

Integrated  Maintenance  Information  System 
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information-systems-orientated 
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